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We describe a practical algorithm which computes the accepting 
automaton for the insertion encoding of a permutation class, when- 
ever this insertion encoding is regular. This algorithm is imple- 
mented in the accompanying Maple package INSENC, which can 
automatically compute the rational generating functions for such 
classes. 



1. Introduction 

Permutation classes, or restricted permutations, have received considerable attention over 
the past two decades, and during this time a great variety of techniques have been used 
to enumerate them. One of the most popular approaches, pioneered by Chung, Graham, 
Hoggatt, and Kleiman [4], employs generating trees. The permutation classes with finitely 
labeled generating trees were characterized in Vatter [14]. A more powerful technique 
based on formal languages and called the insertion encoding was later introduced by Al- 
bert, Linton, and Ruskuc [2]. While they characterized the classes that possess regular 
insertion encodings, naively employing their techniques requires the determinization of 
non-deterministic automata several times, and no implementation has been available. We 
study regular insertion encodings from a new point of view, essentially focusing on ac- 
cepting automata instead of languages. This leads both to an implementation (the Maple 
package INSENC, available for download from the author's homepage) and to a new proof 
of the characterization of permutation classes with regular insertion encodings. 

We begin with definitions. Two sequences of natural numbers are said to be order iso- 
morphic if they have the same pairwise comparisons, so 9, 1, 6, 7, 2 is order isomorphic to 
5, 1, 3, 4, 2. Every sequence w of natural numbers without repetition is order isomorphic 
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to a unique permutation that we denote by st(u;), so st(9, 1, 6, 7, 2) = 5, 1, 3, 4, 2, which we 
shorten to 51342. We call st(w) the standardization of w. We further say that the permuta- 
tion 7r contains the permutation (3 if 7r contains a subsequence that is order isomorphic to 
/?, and in this case we write j3 ^ it. For example, 391867452 contains 51342, as can be seen 
by considering the subsequence 91672. If it does not contain (3, then 7r is said to avoid [3. 

A permutation class is a lower order ideal in the containment ordering, meaning that if 7r 
is contained in a permutation in the class, then ir itself lies in the class. Permutation classes 
can be specified in numerous ways, but we focus solely on the most common method, 
in which the minimal permutations not in the class are given (this set is called the basis). 
By the minimality condition, bases are necessarily antichains, meaning that no element of 
a basis is contained in another. Although there are infinite antichains of permutations 
(see Atkinson, Murphy, and Ruskuc [3] for constructions and references to earlier work), 
we restrict our attention to finitely based classes. Given a set of permutations B, we define 
Av(B) to be the set of permutations that avoid every permutation in B. Thus if C is a closed 
class with basis B then C = Av(B), and for this reason the elements of a permutation class 
are often referred to as restricted permutations. We let C n denote the set of permutations of 
length n in C and refer to J] \C n \x n as the generating function of C. All generating functions 
herein include the empty permutation of length 0. 

2. Finitely Labeled Generating Trees 

In the generating tree approach to enumerating Av(B), the first step is to construct the 
pattern-avoidance tree T(B) in which the children of the permutation ir e Av„_i(l?) are all 
permutations in Av n (B) which can be formed by inserting n into n. The active sites of the 
permutation it e Av n _i(l?) are defined as the indices i for which inserting n immediately 
before 7r(i) produces a 5-avoiding permutation (we also allow for n to be an active site). 
Thus every permutation has as many children in the pattern-avoidance tree as it has active 
sites. A principle subtree in T(B) is a subtree consisting of a single permutation and all of 
its descendants. 

If it happens that every principle subtree in the pattern-avoidance tree T(B) belongs 
to one of a finite number of isomorphism classes, then the (rational) generating func- 
tions for Av(B) can be easily computed using the transfer matrix method (see Flajolet and 
Sedgewick [6, Section V.6]). Following [14], we say that the entry of 7r is generating- 
tree-reducible (relative to B), or simply GT-reducible, if the principle subtree rooted at it and 
the principle subtree rooted at st(7r — 7r(i)) are isomorphic as rooted trees. (Here it — ir(i) 
denotes the sequence obtained by deleting the entry 7r(i) from tt.) We can now state the 
main theorem of Vatter [14]: 

Theorem 1 (Vatter [14]). For a finite set B of permutations, the following are equivalent: 

(1) B contains both a child of an increasing permutation and a child of a decreasing permutation, 

(2) there is an integer k such that no node ofT(B) has more than k children, 
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(3) every sufficiently long permutation in Av(B) contains a GT-reducible entry, 

(4) T(B) has only finitely many isomorphism classes of principle subtrees (in other words, 
Av(B) has a finitely labeled generating tree). 

The implications (3) => (4) and (4) => (2) are trivial, while (1) <^^> (2) follows rou- 
tinely from the Erdos-Szekeres Theorem; the main content of the theorem is that (2) => (3). 

3. The Insertion Encoding 

Before describing the insertion encoding, we briefly review regular languages and finite 
automata. The classic results mentioned here are covered more comprehensively in many 
texts, for example, Flajolet and Sedgewick [6, Appendix A. 7], so we give only the barest 
details. 

A deterministic finite automaton (DFA) M over the alphabet E consists of a set S of states, 
one of which, so, is designated the initial state, a transition function 5 : S x E — > S, and 
a subset A c S designated as accept states. We denote this by M = (S, E, so, A, 5). It is 
useful to extend the definition of the transition function 5 to a map 5 : S x T,* ^ S in the 
obvious way. We say that the state t is reachable from the state s if there is a word w e E* 
such that S(s, w) = t; otherwise, t is unreachable from s. The automaton M accepts the word 
w e E* if 5(sq, w) is an accept state. The set of all such words is the language accepted by the 
automaton, C(M). 

A language that is accepted by a finite automaton (deterministic or not) is called recog- 
nisable. By Kleene's Theorem, the recognisable languages are precisely the regular lan- 
guages, and our purposes the reader may simply take this as the definition of regular lan- 
guages. Regular languages have numerous pleasing properties, but for our purposes we 
only care that they have rational generating functions which may be computed from their 
accepting automata. 

Central to the insertion encoding is the notion of a configuration. Consider, for exam- 
ple, the permutation it = 423615. In the generating tree viewpoint, ir is a descendant of 
4231. In the insertion encoding viewpoint, we note that n is obtained from 4231 by in- 
serting entries between the 3 and the 1 and after the 5. Thus we say that tt evolves from 
the configuration 423 o 1 o. Formally, a configuration is a permutation together with zero 
or more o entries called slots, which may not be adjacent and must eventually be filled. 
Permutations correspond to the slotless configurations. 

Given a configuration, there are four different ways to insert a new maximum entry 
m into the ith slot (we number slots from left to right) of a configuration. We may insert 
this maximum entry into the middle of the slot (replacing the o with omo), to the left of 
the slot (replacing the o with mo), to the right of the slot (replacing the o with om), or we 
may fill the slot (replacing the o with m). These four types of operations are denoted by 
mj, lj, r-j, and fj, respectively. This gives a unique encoding of every permutation, called 
the insertion encoding 1 . For example, the insertion encoding of 423615 is min^^fifbfi- The 



'This encoding actually dates back at least to the work of Francon and Viennot [7], although instead of 
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Figure 1: A vertical parallel alternation (left) and a vertical wedge alternation (right). 



insertion encoding of a permutation class Av(-B) is the language consisting of the insertion 
encodings of every element of Av(B). We say that a configuration is valid for Av(B) if it 
can be filled to produce a permutation in Av(B). 

For any regular language C, there is an integer k such that if w; is a prefix of a word 
in C, then w is a prefix of a word in £ with at most k additional symbols 2 Thus for the 
insertion encoding of Av(B) to be regular, there must be a bound on the number of slots 
in valid configurations for Av(B); if Av(B) satisfies this constraint, we call it slot-bounded. 
Thus Av(B) cannot contain arbitrarily long vertical alternations, which are permutations in 
which every even (resp., odd) indexed entry lies above every odd (resp., even) index entry 
By the Erdos-Szekeres Theorem [5], every long vertical alternation contains a long vertical 
parallel alternation or a long vertical wedge alternation (see Figure 1), which makes it easy 
to check if the insertion encoding of a class needs only a finite alphabet 3 . This necessary 
condition is also sufficient: 

Theorem 2 (Albert, Linton, and Ruskuc [2]). For a finite set B of permutations, the following 
are equivalent: 

(1) Av(B) contains only finitely many vertical alternations, 

(2) there is an integer k such that no valid configuration for Av(B) has more than k slots, 
(4) the insertion encoding ofAv(B) is regular. 



4. Our Approach to the Insertion Encoding 

While Albert, Linton, and Ruskuc considered the insertion encoding from the viewpoint 
of formal languages, our approach parallels that of Theorem 1, and borrows terminology 
from the minimization of DFAs. 

words, they used colored Motzkin paths. 

2 To see this, suppose that the accepting automaton, say (S, E, so, A, 5), for C has k + 1 states. If there is a 
path from 5 (so, w) to an accept state, then there is a path from 8 (so, w) to an accept state which does not revisit 
any states, and thus has length at most k. 

3 More explicitly, Av(B) contains only finitely many vertical parallel alternations oriented as in Figure 1 if 
and only if B contains a member of Av(123, 3142, 3412), while it contains only finitely many vertical wedge 
alternations oriented as in Figure 1 if and only if B contains a member of Av(132, 312). One must also check 
the reversals of these classes, Av(321, 2143, 2413) and Av(213, 231). 
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Given a DFA M = (S, E, sq, A, 5) over the alphabet E with no unreachable states, we 
say that w e E* distinguishes between two states s and £ of M if <5(s, u;) 6 C(M) and 

iy) £(M), or vice versa. Two states are called indistinguishable if there is no word 
which distinguishes them. This defines an equivalence relation on the states of M; for a 
state s e S, we let [s] denote the equivalence class of all states which are indistinguishable 
from s. By the Myhill-Nerode Theorem, this equivalence relation describes the minimal 
automation for C(M), that is, the automaton with the minimum possible number of states 
which accepts the language C(M). 

The Myhill-Nerode Theorem [12, 13]. Let M = (S, E, s , A, 5) be a DFA with no states which 
are unreachable from sq. The DFA M = (S, E, [s ], A, 5) where S = {[s] : s e S}, A = {[s] : s e 
A}, and 5(\s\,w) = [sw] is the minimum DFA for C(M). 

Suppose that we are given the basis B for a slot-bounded class Av(B), and that we 
would like to construct the accepting automaton for the insertion encoding of Av(B). We 
could build an infinite accepting automaton in which the states of the automata are the 
valid configurations for Av(B). The initial state would be o, while the accept states would 
be the slotless configurations, and the transitions would be the obvious transitions given 
by inserting in the middle of a slot, to the left or right, or filling the slot. 

In order to construct a finite automaton which accepts the insertion encoding of Av(B), 
we essentially minimize this infinite automaton, although we focus only on a special type 
of indistinguishable states. We say that the entry c(i) in the configuration c is insertion- 
encoding-reducible (relative to B), or simply IE-reducible if c is indistinguishable from st (c — 
c(i)), where here we have extended the notion of standardization to states in the obvious 
manner, e.g., st(9 o 1672 o) = 5 o 1342 o. Note, trivially, that c(i) will not be IE-reducible 
if it is a o or if has os to both of its sides, as then c(i) and st(c — c(i)) will have a different 
number of slots. 

In the next proposition, which verifies that IE-reducibility is decidable, we say that 
the word w weakly distinguishes the states s and t if one can reach an accept state from 
6(s, w) but not from 5(t, w), or vice versa. Note that a weakly distinguishing word need 
not distinguish the two states, but its existence implies that the states are distinguished by 
some word. 

Proposition 3. Let cbe a valid configuration for Av(B) and suppose that the longest element of 
B has length b. If the entry c(i) is neither a o nor adjacent to two os, then it is IE-reducible if and 
only if no word of length at most b — 1 weakly distinguishes c and st(c — c(i)). 

Proof. If the entry c(i) is IE-reducible then c and st(c — c(i)) are equivalent, so they are not 
weakly distinguished by any words, let alone those of length at most 6 — 1. 

Suppose then that the entry c(i) is not IE-reducible, so c and st(c — c(i)) are distin- 
guished by some word w. If S(c,w) is an accept state then <5(st(c — c(i)),w) must be as 
well, so we may assume that S(c, w) is not an accept state but <5(st(c — c(i)),w) is. Because c 
and st(c — c(i)) have the same number of slots (since c(i) is neither a o nor adjacent to two 
os), this means that 5(c, w) is a permutation, say ir <£ Av(B). Choose some copy of a basis 
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element (3 e B in n. Because st(7r — c(i)) e Av(B), this copy of (3 contains c(i) together with 
at most b — l other entries. By ignoring all entries of tt which are neither in (the underlying 
permutation of) c nor in this copy of (5, we see that there is a word v of length at most b — l 
which weakly distinguishes c from st(c — c(i)), as desired. □ 

We are now ready to state and prove our strengthening of Theorem 2. 

Theorem 2'. For a finite set B of permutations, the following are equivalent: 

(1) Av(-B) contains only finitely many vertical alternations, 

(2) there is an integer k such that no valid configuration for Av(B) has more than k slots, 

(3) every sufficiently long configuration contains an IE-reducible entry, 

(4) the insertion encoding ofAv(B) is regular. 

As with Theorem 1, note that three of the implications in Theorem 2' are trivial. We 
have already remarked that (4) => (2) and (2) <=^> (1), while (3) => (4) because (3) shows 
that the insertion encoding for Av(B) has a finite accepting automaton. Only (2) ==> (3) 
remains. 

Proof that (2) =^> (3) in Theorem 2'. Suppose that the longest element of B has length b. We 
are given that no valid configuration (for Av(B)) has more than k slots for some k, and 
must show that every sufficiently long configuration contains an IE-reducible entry. 

Given a configuration c of length n, let / c [n] denote the set of indices such that c(i) 
is neither a o nor adjacent to a o. Note that since no valid configuration has more than k 
slots, all but a bounded number of indices lie in I. 

If c(i) is not IE-reducible for some i e I, then the proof of Proposition 3 shows that there 
is a word u of length at most b — l which leads to a valid configuration from st(c — c(i)) 
but not from c. Since <5(st(c — c(i)),u) is a valid configuration with at most k states, there 
must be a word v of length at most k which leads from this configuration to a valid slotless 
configuration, i.e., a permutation in Av(B). Therefore, the word w = uv leads from st(c — 
c(i)) to a permutation in Av(B), while it leads to an invalid configuration from c. We call 
w a witness for i. Every occurrence of an element of B in 5(c, w) must include the entry 
c(i), and so no word may witness more than b elements of /. Therefore, since there are a 
bounded number of possible witnesses (they can each have length at most b + k — 1), each 
can witness at most b indices of /, and / contains all but a bounded number of indices, for 
n sufficiently large, there must be at least one index of I without a witness. Clearly this 
witness-less element of / is IE-reducible, completing the proof. □ 

It follows easily from the definitions that if tt{i) is GT-reducible (relative to B) for the 
permutation tt, then in every configuration whose underlying permutation is n, the entry 
corresponding to is IE-reducible. This verifies that every class with a finitely labeled 
generating tree also has a regular insertion encoding. 
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5. Counting Sum Indecomposable Permutations 

The permutation ir of length n is sum indecomposable (or, connected) if there is no integer 
2 < % < n - 1 such that vr({l, 2, ...,«}) = {1, 2, ... , i}. 

As we describe first, it is fairly easy to characterize the sum indecomposable permuta- 
tions via the insertion encoding. The evolution of a sum decomposable permutation must 
contain a non-initial configuration whose only slot occurs at the end of the configuration. 
Conversely, every permutation which can be formed from such a configuration is sum de- 
composable. If Av(-B) has a regular insertion encoding, then we know by Theorem 2 that 
there is a constant k such that no valid configuration for Av(-B) has more than k slots. In 
order to recognize the sum indecomposable permutations in this class, we therefore need 
only to keep track of how many open slots a configuration has and whether the rightmost 
slot occurs at the end of the configuration, rejecting a permutation whenever its evolution 
includes a non-initial configuration whose only slot occurs at the end of the configuration. 
It follows from the closure properties of regular languages that the sum indecomposable 
permutations in a class with a regular insertion encoding also have a regular insertion 
encoding. 

While this shows that the sum indecomposable permutations in a class Av(B) with 
a regular insertion encoding themselves have a regular insertion encoding, it describes 
a rather circuitous route to this encoding. Instead, a straight-forward adaptation of our 
approach leads directly to the accepting automaton for the insertion encoding of sum in- 
decomposable permutations in Av(B). Let us say that the element c(i) of the configuration 
c is SIE-reducible (relative to B) if it is IE-reducible and is not the rightmost entry of c. It 
follows from our proof of Theorem 2' that every sufficiently long valid configuration for 
Av(B) has more than one IE-reducible entry, and so has at least one SIE-reducible configu- 
ration. To construct the accepting automaton for the sum indecomposable permutations in 
Av(B), one therefore eliminates all configurations whose only slot occurs at the end, and 
identifies c and st(c — c(i)) whenever c(i) is SIE-reducible. This is also implemented in the 
package INSENC. 

6. Conclusion 

We have presented a new viewpoint of regular insertion encodings, which has lead to a 
new proof of Theorem 2 and to an implementation in the Maple package INSENC, available 
from the author's homepage. We conclude with some results obtained from this package. 

Recall that the three permutation class symmetries inverse (ir h- > it -1 ), reverse (n >— > 
7r(n) ■ ■ ■ 7r(2)7r(l)), and complement (ir h- > (n + 1 — 7r(l))(n + 1 — vr(2)) • • • (n + 1 — n(n))) 
generate the symmetries of the square. Given a set of permutation classes, it is therefore 
useful to divide them into symmetry classes. For example, the (g) = 276 permutation 
classes with precisely two basis elements of length 4 fall into 56 distinct symmetry classes. 
Two permutation classes are further said to be Wilf-equivalent if they are equinumerous. 
Le [11] recently established that these 56 symmetry classes form 38 distinct Wilf classes. Of 
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these 38, 12 can be enumerated with regular insertion encodings. Of those 12, 10 of those 
can be enumerated using finitely labeled generating trees, and their generating functions 
are reported in Vatter [14] (these generating functions were also computed by hand by 
Kremer and Shiu [10]). The 2 new generating functions are listed below. 



Class 


Generating function 


Av(4321, 1324) 
Av(4321, 3142) 


l-llz+56x 2 -172z 3 +357x' 4 -519a.- 5 +554x 6 -413a; 7 +217x 8 -83x 9 +20x 10 -2a; 11 


(l-x)(l-3x) 2 


(l-2x)' 2 (l-Ax+x 2 ) 



From the generating function displayed above, it follows that for large n, the number 
of permutations in Av(4321, 1324) of length n is given by a polynomial. This is not a sur- 
prise, as it can be checked that this class meets the conditions of Huczynska and Vatter [8] 
or Albert, Atkinson, and Brignall [1] who, building on the work of Kaiser and Klazar [9], 
characterized the permutation classes of polynomial growth. In fact, it follows from Theo- 
rem 2 that all permutation classes with lower growth rate 4 less than 2 have regular inser- 
tion encodings. 

Acknowledgments: The author thanks Michael Albert for numerous helpful suggestions, 
and for pointing out the work of Francon and Viennot [7]. 
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